Skip to content

fix(zombie): cap zombie detection by the latest send time (0.7.3, hot-patchable) - #72

Merged
zmstone merged 2 commits into
patch-0.7.3from
260925-0.7.3-hotpatch-zombie-cap
Sep 25, 2026
Merged

zmstone merged 2 commits into
patch-0.7.3from
260925-0.7.3-hotpatch-zombie-cap

Conversation

@zmstone

@zmstone zmstone commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Refs #70. A hot-patchable variant of #71 for patch-0.7.3.

Why

Zombie-connection detection fires at request timeout + max_inactive after the latest send. A production deployment runs with a 600 s request timeout, so a dead connection is reconnected only after 610 s. Requests with an infinity timeout never record an expiry, so detection never fires for them.

What

  • put_sent_req/3 records the send time in the process dictionary under last_sent_ts.
  • check_gun_jammed/3 reads last_sent_ts. When it is an integer and requests are in flight, the connection is also reconnected when now - last_sent_ts > max(max_inactive, ?INACTIVE_CAP). ?INACTIVE_CAP is 60 s.
  • When last_sent_ts is undefined, only the existing expiry check runs.
  • The existing expiry check always runs, so detection is never later than before.

The combined rule is the clamp from #71, max(MaxInactive, min(Cap, Timeout + MaxInactive)) after the latest send, written without a stored timeout:

max_inactive request timeout detection before detection after
10s 5s 15s 15s
10s 45s 55s 55s
10s 600s 610s 60s
10s infinity never 60s
180s 600s 610s 180s

Hot patch

#state{}, the requests map and all message formats are unchanged. The module can be loaded into a running node with code:load_file/1 or c:nl/1, and no code_change is needed. A worker that has not sent a request since the load has no last_sent_ts, so it uses only the expiry check until its next send.

The reconnect branch in maybe_shoot/1 no longer asserts MaxExpire > 0, because a pool of infinity-timeout requests now reaches it with no expiry recorded. The zombie log adds last_request_sent_at and inactive_duration_cap.

Incompatible case

A response that legitimately takes longer than max(max_inactive, 60s) now has its connection killed at that point. Received data does not count as activity. Raise max_inactive to the slowest expected response.

Tests

?INACTIVE_CAP is 2 s in the TEST build. With the default max_inactive of 10 s the effective cap stays 10 s, so the other tests are not affected.

  • zombie_detect_long_timeout_test: max_inactive = 1s, request timeout 30 s, server never answers. reconnect fires between 2 s and 3.5 s.
  • zombie_detect_infinity_timeout_test: the same, with request timeout infinity.
  • The existing zombie_detect_inflight_* tests pass unchanged.

Both new tests fail against unpatched patch-0.7.3.

Run locally on OTP 28: make fmt-check, ./check-style.sh, make compile, make xref, make dialyzer, make eunit. make eunit has 9 failures, all in ehttpc_google_tests:proxy_test_, because tinyproxy is not installed on the host. The hot load itself was not tested on a live node.

CI

.github/workflows/erlang.yml ran only for main, so this PR got no CI run. A separate ci: commit adds 'patch-*' to the push and pull_request branch filters.

Zombie detection fired at request timeout + max_inactive after the
newest send, so it scaled with the caller's timeout. With a 600 s
request timeout a dead connection was reconnected only after 610 s.
Requests with an infinity timeout never recorded an expiry, so
detection never fired for them.

put_sent_req/3 now records the send time in the process dictionary
under last_sent_ts. When it is set and requests are in flight,
check_gun_jammed/3 also reconnects when the latest send is older than
max(max_inactive, ?INACTIVE_CAP), with the cap at 60 s. The existing
expiry check stays, so detection is never later than before.

The send time lives in the process dictionary, not in #state{} or the
requests map, so the worker state and message formats do not change.
The module can then be hot-patched by loading the beam into a running
node. A worker that has not sent since the load has no last_sent_ts and
uses only the expiry check.

The reconnect branch no longer asserts that an expiry was recorded,
because a pool of infinity-timeout requests now reaches it with none.
The zombie log adds the latest send time and the cap.

Under TEST the cap is 2 s, so the tests exercise it in seconds.

Refs #70
@zmstone
zmstone marked this pull request as ready for review September 25, 2026 19:51
@zmstone
zmstone changed the base branch from 0.7.3-branch to patch-0.7.3 September 25, 2026 19:56
The workflow ran only for pushes to and pull requests against main,
so pull requests against patch branches such as patch-0.7.3 got no
CI run. Match patch-* as well.
@zmstone
zmstone force-pushed the 260925-0.7.3-hotpatch-zombie-cap branch from 2cf696c to 29d9cec Compare September 25, 2026 19:57
Comment thread src/ehttpc.erl
-ifdef(TEST).
-define(INACTIVE_CAP, 2_000).
-else.
-define(INACTIVE_CAP, 60_000).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since max_inactive is always finite, instead of hardcoding the limit at 60 s, would it make sense to use max_inactive as the cap in max_expire when the request ttl is infinity?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the problem is max_inactive might be too small.
e.g. max_inactive is set to 5s (real config from the reporter), for large payload, if it takes longer than 5s to send, then the request is doomed to expire.

@zmstone
zmstone merged commit a41658f into patch-0.7.3 Sep 25, 2026
3 checks passed
@zmstone
zmstone deleted the 260925-0.7.3-hotpatch-zombie-cap branch September 25, 2026 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants